Students, journalists, secretaries and others often attend lectures, meetings and other events and need to remember the information presented. Usually they take notes by hand. They may use these notes as they are, or enter them into a word processor later. Unfortunately, because a speaker, teacher or lecturer can present information much faster than it can be written, notes taken during a lecture or meeting are typically incomplete and often contain inaccuracies.
Another disadvantage of taking notes is that it conflicts with full participation and interaction with other people present. It can be difficult to make comments and ask or answer questions while taking notes.
Also, the information that is presented is better received when the listener gives undivided thought. The need to take notes competes with concentration on the material itself. When the material is complex, taking notes can interfere with the depth of concentration required to comprehend what is being presented. The material will then have to be learned later, often from the incomplete or inaccurate notes.
In an attempt to overcome these problems, some people make an audio recording for later reference. An audio recording can solve the problems of omissions and inaccuracies. But an audio recording is accessed sequentially. It is most suited for listening to the entire event or a large portion of it a second time. The recording can not provide a summary of the event, and it is difficult to pick out its most important parts. It can not be reviewed as quickly and readily as some notes on a few sheets of paper can be reviewed. If there is a question about the accuracy of a particular point in the notes, or an omission in the notes, it is difficult to verify or fill in the omission without a timeconsuming search of the recording. Correcting several such questions or omissions easily takes as long as a second listening of the entire recording. Thus, the recording introduces new problems of accessing the recorded material in a reasonable time.
Another way some people try to take better notes is to use a personal portable computer in the field. The person can enter notes directly into a portable computer as an event occurs. The computer software used for this purpose may be a word processor, text editor, memo field or the like. For users who can type fast enough, this can be an improvement over handwritten notes. The new notes can be added to a file in the computer that containing all the notes of a similar topic, such as previous class meetings of a college course. Using a computer, it is easy to make the notes more coherent and presentable. Thus, using a computer in this manner can be beneficial.
However, using a portable computer to take notes of an event as it occurs does not eliminate the problem of omissions and inaccuracies in the notes. For instance, although experienced typists can type text faster than it can be written in longhand, they can not type exact quotations spoken at normal speed. Taking notes becomes even harder when the subject matter is unfamiliar, complex or spoken quickly. Important information or exact wording can be lost or mistranscribed. Further, under most circumstances it is difficult to operate the computer keyboard under such intense circumstances and at the same time pay attention to, respond to, or otherwise interact with the speaker. Additionally, intense keyboard clatter can annoy other people who are present.
Most portable computers have multimedia hardware that enables them to make and play back an audio recording. Such portable multimedia computers can record events lasting from minutes to many hours, depending on the amount of computer storage available (typically a hard disk drive or RAM). Software to help in this task is available. Typically, such software is modeled after a tape recorder paradigm. It presents buttons to the user such as xe2x80x9cPlay,xe2x80x9d xe2x80x9cStop,xe2x80x9d xe2x80x9cPause,xe2x80x9d xe2x80x9cFast Forward,xe2x80x9d xe2x80x9cRewind,xe2x80x9d and xe2x80x9cRecord.xe2x80x9d Some additionally have a slider for immediate access to any part of the recording without the need to fast forward or rewind. The slider is moved using a pointing device such as a mouse, track ball or track pad, touch screen, or by keystrokes.
However, regardless of whether a tape recorder or the computer itself is used to record the audio event, there remains the problem of having to later locate on the tape, or the computer""s recording (typically a file on the computer hard disk), the pertinent recorded portions. As is well known, such a process is usually done by trial and error: Usually the user fast forwards and rewinds the recording until the needed portion is located. On some multimedia computers, the search may be done by using a pointing device (such as a mouse) to move a slider control back and forth until the desired part is found.
One of the difficulties of a trial and error search is that upon hearing a portion of the recording, it is often difficult to determine whether it is before or after the desired portion, and by how much. Thus, it is necessary to listen to each portion long enough to determine the subject matter at that point and to remember or guess where the portion fits in relative to the desired portion. This is difficult to do when the lecture or other audio event is unfamiliar, which is usually the case. The whole process discourages people from recording something unless they esteem it important enough to invest considerable time into listening to all of it or at least much of it again. Often recordings made with good intentions are never listened to at all.
Sometimes an exact transcript of an event is important enough to hire a stenographer to create it. U.S. Pat. No. 5,272,571 to Henderson et al discloses a stenotype machine and computer system for use by people skilled in stenotype, such as court reporters. A stenotype operator transcribes spoken words into their phonetic representations by entering phonetic keystrokes on a stenotype machine. According to Henderson, as the stenotype operator does this, the stenotype machine makes a digital audio recording and establishes links (pointers) between the phonetic keystrokes and the digital audio recording. Later, these data are transferred to a computer. The computer attempts to look up the phonetic keystrokes in dictionaries of such keystrokes to transcribe the phonetic keystrokes into text. Phonetic keystrokes that are not found in the dictionary are known as xe2x80x9cuntranslates.xe2x80x9d The court reporter can resolve the untranslates and anything else that might be mistranslated by directing the computer to play back the relevant audio portion. The computer finds the link between the untranslated (or any phonetic keystroke) and the audio recording. It uses this link as a pointer into the audio recording to play back the particular audio recorded proximate to those phonetic keystrokes.
While this stenotype system may be useful for its intended purpose, where phonetic keystrokes are entered in step with each syllable spoken, it does not meet the needs of students and others who merely want to take notes on a word processor. One reason is that most people have not acquired the skill of operating a stenotype machine. Operating a stenotype machine requires extensive training and enough skill to keep pace with the speaker. On the other hand, virtually everyone can operate an ordinary typewriter or computer keyboard at some speed, fast or slow, even without training.
Another reason a stenotype system does not meet the needs of people who want to take notes is that they usually do not need an entire transcript. The person taking notes on paper or into a word processor is free to summarize, insert personal comments or otherwise change the information presented. Taking notes into a word processor is usually at least partially a creative endeavor. Nevertheless, the information often comes too fast to handle in such a creative endeavor and important facts are missed. Even when there is adequate time to enter the notes, the notes are not entered in sync with a speaker.
Therefore, there is need for something that helps a person who take notes, possibly including personal comments, sketches, exact quotations or even an entire transcript, to fix these notes in some tangible form with reduced risk of inaccuracies or loss, regardless of the disparity between the pace of a presentation and the person""s ability to write or type, and to access and use this information in a convenient manner. The present invention addresses this need.
1. Objects
It is an object of the invention to provide a method of using a computer with multimedia capability to improve the note-taking capabilities of individuals who want to take accurate notes. It is also an object of the invention to provide a way for users to take notes that include exact quotations, ranging from an occasional quote up to an entire transcript, as they wish, that the users could not otherwise accurately and conveniently include in their notes. It is a further object of the invention to greatly reduce or often eliminate the need to search by trial and error for a particular portion of an audio recording.
It is a further object of the invention to allow users to conveniently augment their notes after an audio event, such that whenever they suspect they may have missed something in their notes, they can quickly and conveniently review exactly what was said or shown and supplement their notes as desired.
It is a further object of the invention to free users of the invention to more fully participate at lectures, meetings, and other circumstances by greatly reducing the effort needed to take notes during these events. Another object is to enable people who use notebook computers to take notes to reduce the amount of keyboard noise they create without reducing the quality of their notes.
A further object of the invention is to provide a method of programming a computer that allows a user to utilize the audio recording capabilities of computer hardware in conjunction with the textual input capabilities of a computer word processor in an integrated and seamless fashion.
A still further object of the invention is to provide a way to access particular points in an audio recording by reference to various portions of text in a word processor that reveal the significance of each point in the audio recording.
It is also an object of the invention to provide the above objects in highly portable computing devices, including, without limitation, notebook computers, subnotebook computers, palm computers, personal digitalal assitants, and so on.
2. Summary
The foregoing and other objects are achieved with a novel method of controlling the multimedia capabilities of a computer and a word processor or similar text processing application, and integrating the word processor and multimedia capabilities of the computer. The invention enables users to utilize a new kind of word processor to enter notes while listening to a lecture, conversation, presentation or similar event and simultaneously use the computer""s multimedia capability to make an audio recording of the event. The invention also includes video recording for computers that can provide this function.
The word processing and multimedia functions are integrated in that as the user enters text into the word processor, the portion of the text being entered is linked to the audio portion that is being recorded. This is done by semi-automatically and/or automatically establishing xe2x80x9caudio linksxe2x80x9d (or just xe2x80x9clinksxe2x80x9d) that associate particular points in the text with particular points in the audio recording. When an audio event is over, the text contains notes of the important parts of the event, complete or incomplete. These notes are linked directly to the corresponding portions of the audio recording. The method also accommodates the entry of text and establishment of audio links while playing a recording of a previously-recorded audio event.
The word processing and audio functions are further integrated for playback of the audio recording. The user can use the keyboard or a pointing device to control the playback of the audio recording. This is done by selecting a portion of the text and telling the computer to play back the portion of the audio recording that is linked to the selected text. The audio portion played back is the audio portion that was recorded when the selected text was originally entered during the audio event. The user can play-back the entire audio event or any portion of it, depending on how much text is selected. The user can choose to never play back portions of the audio recording linked to notes that are satisfactorily complete and play back only those portions that are linked to notes that are incomplete or suspect. The user does not have to xe2x80x9cscanxe2x80x9d the recording or search it by trial and error in order to locate the desired audio portion(s). The user need only select that portion of the text that corresponds to the desired audio portion.
The invention also supports the display of audio links within the text in several ways. The user can point to such a display with a pointing device such as a mouse and click on the display. The invention will then look up the audio link and start playing the audio starting from the moment recorded when the audio link was first created. Some preferred ways an audio link can be displayed include, without limitation, highlighting a word in the text so it looks like a HTML-style hyperlink (such as in a different color and/or underlined), displaying a special symbol that is inserted in the text, displaying special indicia along the left or right margin of the text, and displaying the time of day the link was created in the left or right margin of the text.
Another feature of the invention is to allow replay of the audio a segment at a time, either for predetermined segment lengths or for as long as a button is held down. The invention provides that the user can specify an overlap time, so that, for example, a spoken word that is cut off at the end of one segment will be fully played at the beginning of the next segment.
The invention also provides optional features that allow the user to adjust playback speed and/or pitch, and to protect against the invention suddenly playing audio aloud at an inappropriate time, such as during or just after recording an event.
The invention also provides for outputing the text and multimedia data it generates to both standard HTML and an enhanced form of HTML that can specify playing an audio file starting at any arbitrary point.
An optional feature of the invention provides the user with the ability to selectively delete portions of an audio record by indicating on the text which portions of the audio recording are to be deleted, and thereby free up audio storage space.
Another optional feature of the invention for minimizing the use of audio storage space, when activated by the user, causes the computer to detect when a portion of an event is of no interest to the user and to discard that portion of the audio recording, and when a following portion of the event becomes of interest to the user to retain a portion of the recording for a predetermined amount of time just prior to when the user decided the following portion was of interest, as determined by express command or automatically by keyboard activity.
Another feature of the invention is that the recording and playback of the audio event is controlled according to a set of xe2x80x9cuser preferences.xe2x80x9d The user preferences determine when to establish audio links, how to compensate for delays, how replay is controlled, which features are in effect at any time, and other matters. Different sets of user preferences are conveniently available for different users and to adapt the invention to different types of audio events.